{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Guided Project 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning Objective:**\n",
    "\n",
    "* Learn how to customize the tfx template to your own dataset\n",
    "* Learn how to modify the Keras model scaffold provided by tfx template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this guided project, we will use the `tfx template` tool to create a TFX pipeline for the covertype project, but this time, instead of re-using an already implemented model as we did in guided project 2, we will adapt the model scaffold generated by `tfx template` so that it can train on the covertype dataset\n",
    "\n",
    "**Note:** The covertype dataset is loacated at\n",
    "```\n",
    "gs://workshop-datasets/covertype/small/dataset.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Environment setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Envirnonment Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the your Kubeflow pipelines endopoint below the same way you did in guided project 1 & 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ENDPOINT = # Enter your Kubeflow ENDPOINT here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH=%env PATH\n",
    "%env PATH={PATH}:/home/jupyter/.local/bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "GOOGLE_CLOUD_PROJECT=shell_output[0]\n",
    "\n",
    "%env GOOGLE_CLOUD_PROJECT={GOOGLE_CLOUD_PROJECT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Docker image name for the pipeline image.\n",
    "CUSTOM_TFX_IMAGE = 'gcr.io/' + GOOGLE_CLOUD_PROJECT + '/tfx-pipeline'\n",
    "CUSTOM_TFX_IMAGE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `tfx` and `kfp` tools setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "TFX_PKG=\"tfx==0.22.0\"\n",
    "KFP_PKG=\"kfp==0.5.1\"\n",
    "\n",
    "pip freeze | grep $TFX_PKG || pip install -Uq $TFX_PKG\n",
    "pip freeze | grep $KFP_PKG || pip install -Uq $KFP_PKG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart the kernel at this point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `skaffold` tool setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "LOCAL_BIN=\"/home/jupyter/.local/bin\"\n",
    "SKAFFOLD_URI=\"https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64\"\n",
    "\n",
    "test -d $LOCAL_BIN || mkdir -p $LOCAL_BIN\n",
    "\n",
    "which skaffold || (\n",
    "    curl -Lo skaffold $SKAFFOLD_URI &&\n",
    "    chmod +x skaffold               &&\n",
    "    mv skaffold $LOCAL_BIN\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify the `PATH` environment variable so that `skaffold` is available:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, you shoud see the `skaffold` tool with the command `which`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!which skaffold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Copy the predefined template to your project directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, we will create a working pipeline project directory and \n",
    "files by copying additional files from a predefined template.\n",
    "\n",
    "You may give your pipeline a different name by changing the PIPELINE_NAME below. \n",
    "\n",
    "This will also become the name of the project directory where your files will be put."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PIPELINE_NAME = # Your pipeline name\n",
    "PROJECT_DIR = os.path.join(os.path.expanduser(\".\"), PIPELINE_NAME)\n",
    "PROJECT_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TFX includes the taxi template with the TFX python package. \n",
    "\n",
    "If you are planning to solve a point-wise prediction problem,\n",
    "including classification and regresssion, this template could be used as a starting point.\n",
    "\n",
    "The `tfx template copy` CLI command copies predefined template files into your project directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx template copy \\\n",
    "  --pipeline-name={PIPELINE_NAME} \\\n",
    "  --destination-path={PROJECT_DIR} \\\n",
    "  --model=taxi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd {PROJECT_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3. Browse your copied source files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The TFX template provides basic scaffold files to build a pipeline, including Python source code,\n",
    "sample data, and Jupyter Notebooks to analyse the output of the pipeline. \n",
    "\n",
    "The `taxi` template uses the same Chicago Taxi dataset and ML model as \n",
    "the [Airflow Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/airflow_workshop).\n",
    "\n",
    "Here is brief introduction to each of the Python files:\n",
    "\n",
    "`pipeline` - This directory contains the definition of the pipeline\n",
    "* `configs.py` — defines common constants for pipeline runners\n",
    "* `pipeline.py` — defines TFX components and a pipeline\n",
    "\n",
    "`models` - This directory contains ML model definitions.\n",
    "* `features.py`, `features_test.py` — defines features for the model\n",
    "* `preprocessing.py`, `preprocessing_test.py` — defines preprocessing jobs using tf::Transform\n",
    "\n",
    "`models/estimator` - This directory contains an Estimator based model.\n",
    "* `constants.py` — defines constants of the model\n",
    "* `model.py`, `model_test.py` — defines DNN model using TF estimator\n",
    "\n",
    "`models/keras` - This directory contains a Keras based model.\n",
    "* `constants.py` — defines constants of the model\n",
    "* `model.py`, `model_test.py` — defines DNN model using Keras\n",
    "\n",
    "`beam_dag_runner.py`, `kubeflow_dag_runner.py` — define runners for each orchestration engine\n",
    "\n",
    "\n",
    "**Running the tests:**\n",
    "You might notice that there are some files with `_test.py` in their name. \n",
    "These are unit tests of the pipeline and it is recommended to add more unit \n",
    "tests as you implement your own pipelines. \n",
    "You can run unit tests by supplying the module name of test files with `-m` flag. \n",
    "You can usually get a module name by deleting `.py` extension and replacing `/` with `..`\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m models.features_test\n",
    "!python -m models.keras.model_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. Create the artifact store bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You probably already have completed this step in guided project 1, so you may\n",
    "may skip it if this is the case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Components in the TFX pipeline will generate outputs for each run as\n",
    "[ML Metadata Artifacts](https://www.tensorflow.org/tfx/guide/mlmd), and they need to be stored somewhere.\n",
    "You can use any storage which the KFP cluster can access, and for this example we\n",
    "will use Google Cloud Storage (GCS).\n",
    "\n",
    "Let us create this bucket if you haven't created it in guided project 1.\n",
    "Its name will be `<YOUR_PROJECT>-kubeflowpipelines-default`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GCS_BUCKET_NAME = GOOGLE_CLOUD_PROJECT + '-kubeflowpipelines-default'\n",
    "GCS_BUCKET_NAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gsutil ls gs://{GCS_BUCKET_NAME} | grep {GCS_BUCKET_NAME} || gsutil mb gs://{GCS_BUCKET_NAME}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3. Ingest the data into the pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We made a TFX pipeline for a model using the Chicago Taxi dataset and the covertype dataset. Now it's time to put your data into the pipeline.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your data can be stored anywhere your pipeline can access, including GCS, or BigQuery. You will need to modify the pipeline definition to access your data.\n",
    "\n",
    "Review the steps in guided project 1 and guided project 2 to remember what needs to be customized in full details. You'll find below a short summary of these steps:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. If your data is stored in files, modify the `DATA_PATH` in `kubeflow_dag_runner.py` and set it to the location of your files. If your data is stored in BigQuery, modify `BIG_QUERY_QUERY` in `pipeline/configs.py` to correctly query for your data.\n",
    "1. Add features in `models/features.py`\n",
    "1. Modify models/preprocessing.py to [transform input data for training](https://www.tensorflow.org/tfx/guide/transform).\n",
    "1. Modify `models/keras/model.py` and `models/keras/constants.py` to [describe your ML model](https://www.tensorflow.org/tfx/guide/trainer).\n",
    "  * You can use an estimator based model, too. Change `RUN_FN` constant to `models.estimator.model.run_fn` in `pipeline/configs.py`.\n",
    "1. Modify the `pipeline.py` and `configs.py` so that you can train and deploy on CAIP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Exercise (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a TFX pipeline as we did in this guided project but this time with your own dataset instead of the covertype dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "<font size=-1>Licensed under the Apache License, Version 2.0 (the \\\"License\\\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.</font>"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m50",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m50"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
